On the infeasibility of modeling polymorphic shellcode Re-thinking the role of learning in intrusion detection systems

Current trends demonstrate an increasing use of polymorphism by attackers to disguise their exploits. The ability for malicious code to be easily, and automatically, transformed into semantically equivalent variants frustrates attempts to construct simple, easily verifiable representations for use in security sensors. In this paper, we present a quantitative analysis of the strengths and limitations of shellcode polymorphism, and describe the impact that these techniques have in the context of learning-based IDS systems. Our examination focuses on dual problems: shellcode encryption-based evasion methods and targeted “blending” attacks. Both techniques are currently being used in the wild, allowing real exploits to evade IDS sensors. This paper provides metrics to measure the effectiveness of modern polymorphic engines and provide insights into their designs. We describe methods to evade statistics-based IDS sensors and present suggestions on how to defend against them. Our experimental results illustrate that the challenge of modeling self-modifying shellcode Editors: Pavel Laskov and Richard Lippmann. This material is based on research sponsored by the Air Force Research Laboratory under agreement number FA8750-06-2-0221. Army Research Office contract number W911NF0610151, and by NSF Grant 06-27473, with additional support from Google. Y. Song (!) · A.D. Keromytis · S.J. Stolfo Department of Computer Science, Columbia University, New York, NY 10027, USA e-mail: [email protected] A.D. Keromytis e-mail: [email protected] S.J. Stolfo e-mail: [email protected] M.E. Locasto · A. Stavrou Department of Computer Science, George Mason University, Fairfax, VA 22030, USA M.E. Locasto e-mail: [email protected] A. Stavrou e-mail: [email protected] 180 Mach Learn (2010) 81: 179–205 by signature-based methods, and certain classes of statistical models, is likely an intractable problem.